Stochastic Search In Changing Situations

Stochastic search algorithms are black-box optimizer of an objective function. They have recently gained a lot of attention in operations research, machine learning and policy search of robot motor skills due to their ease of use and their generality. However, when the task or objective function slightly changes, many stochastic search algorithms require complete re-learning in order to adapt thesolution to the new objective function or the new context. As such, we consider the contextual stochastic search paradigm. Here, we want to find good parameter vectors for multiple related tasks, where each task is described by a continuous context vector. Hence, the objective function might change slightly for each parameter vector evaluation. In this paper, we investigate a contextual stochastic search algorithm known as Contextual Relative Entropy Policy Search (CREPS), an information-theoretic algorithm that can learn from multiple tasks simultaneously. We show the application of CREPS for simulated robotic tasks. Introduction Stochastic search algorithms are gradient-free black-box optimizers of some performance function dependent on a highdimensional parameter vector. They directly evaluate the execution of a parameter vector by using the return of an episode. Stochastic search algorithms (Hansen et al. 2003; Sun et al. 2009; Stulp and Sigaud 2012; Rückstieß et al. 2008) typically maintain a search distribution over the parameters that we want to optimise, which is used to create samples of the parameter vector. Subsequently, the performance of the sampled parameters is evaluated. Using the samples and their evaluations, a new search distribution is computed by computing gradient based updates (Sun et al. 2009; Rückstieß et al. 2008), evolutionary strategies (Hansen et al. 2003), the cross-entropy method (Mannor et al. 2003), path integrals (Stulp and Sigaud 2012; Theodorou et al. 2010), or information-theoretic policy updates (Kupcsik et al. 2013; Abdolmaleki et al. 2015a). However, many of the previously mentioned algorithms cannot be applied to multi-task learning. In other words, if the task setup or objective function changes slightly, relearning is needed to adapt the solution to the new situation or the new context. For example, consider optimisCopyright c 2017, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. ing the parameters of a humanoid robot controller to kick a ball. Once the characteristics of the ball, such as weight or material, or objective function, such as desired kick distance, change, re-learning is needed. One could independently optimize for several target contexts in order to generalize a task, for example optimizing to kick the ball for different distances(context). Subsequently, when a new unseen context is presented, the optimized contexts can be generalized through regression methods (Niehaus et al. 2007; Wang et al. 2009). However now optimizing for different contexts and then generalizing between the optimized parameters for different unseen contexts are two independent processes. Therefore, even though such approaches have been used successfully, they are time consuming as well as inefficient in terms of the number of needed training samples. In other words, we cannot reuse data-points obtained from optimizing a task with context s to improve and accelerate the optimization of a task with context s0. As such, it is desirable to learn the selection of the parameters for multiple tasks at once without restarting the learning process once we see a new task. This problem setup is also known as contextual policy search (Kupcsik et al. 2013; Kober et al. 2010). Recently, such multi-task learning capability was established for information-theoretic policy search algorithms (Peters et al. 2010), such as the episodic Contextual Relative Entropy Policy Search (CREPS) algorithm (Daniel et al. 2012; Kupcsik et al. 2013). In (Abdolmaleki et al. 2015c), CREPS was successfully used to optimize a walking controller for different speeds. Despite its advantages, CREPS has a major set-back that does not allow it to perform favourably. Like many other stochastic search algorithms, CREPS maintains a Gaussian search distribution and it updates the mean and covariance matrix of its search distribution iteratively. However due to the covariance matrix update rule of CREPS, we will show that, search distribution might collapse prematurely to a point-estimate before finding a good solution, resulting in a premature convergence which is highly undesirable. Although, this multi-task learning capability is not found in other stochastic search algorithms (Hansen et al. 2003; Sun et al. 2009), such as CMA-ES and NES, or commonly used policy search methods (Stulp and Sigaud 2012; Kober and Peters 2010), they typically don’t suffer from premature convergence. Therefore, to solve premature conPRELIMINARY VERSION: DO NOT CITE